Personal computers (PCs) have become increasingly more powerful in many different respects. One example of the increased power of computers is in their tremendously improved graphics capabilities. While early PCs were limited to four colors and pixilated low resolution displays, contemporary computers provide colorful, high-resolution graphics that are more than suitable for viewing digital photographs or watching movies as well as enabling display of fast moving virtual images in games and other applications.
The improved power of computers also has resulted in today's computers being far more user friendly than their predecessors. Not long ago, personal computers were command-driven, requiring users to remember and enter combinations of keystrokes to direct a computer to perform even simple commands. Today, users engage computers using pointing devices, handwriting recognition, speech recognition, and other simple, intuitive techniques. Personal computers appear on nearly every desk in the workplace. Many households now have multiple computers, and even in-home local area networks.
As computers become more powerful and more ubiquitous throughout our environment, the desire to make computers and their interfaces even more user friendly continues to promote development in this area. For example, the MIT Media Lab, as reported by Brygg Ullmer and Hiroshi Ishii in “The metaDESK: Models and Prototypes for Tangible User Interfaces,” Proceedings of UIST 10/1997:14-17,” has developed another form of “keyboardless” human-machine interface. The metaDESK includes a generally planar graphical surface that not only displays computing system text and graphic output, but also receives user input by responding to an object placed against the graphical surface. The combined object responsive and display capability of the graphical surface of the metaDESK is facilitated using IR (IR) light, an IR camera, a video camera, a video projector, and mirrors disposed beneath the surface of the metaDESK. The mirrors reflect the graphical image projected by the projector onto the underside of the graphical display surface to provide images that are visible to a user from above the graphical display surface. The IR camera can detect IR light reflected from the undersurface of an object placed on the graphical surface.
Others have been developing similar keyboardless interfaces. For example, papers published by Jun Rekimoto of the Sony Computer Science Laboratory, Inc., and associates describe a “HoloWall” and a “HoloTable” that display images on a surface and use IR light to detect objects positioned adjacent to the surface.
By detecting a specially formed object or IR-reflected light from an object disposed on a graphical display surface, the metaDESK can respond to the contemporaneous placement and movement of the object on the display surface to carryout a predefined function, such as displaying and moving a map of the MIT campus. Such systems are generally limited to responding to a specific object in a predefined manner.
It would be desirable to expand upon the functionality of an interactive display system, to enable a user to interact with a display surface more intuitively, naturally, and completely. Ideally, a user should be able to engage a computer system, such as by responding to prompts, issuing commands, or changing attributes, without having to use a keyboard or make use of any specific physical objects. To make the use of a personal computer even more convenient, it would clearly be desirable to interact with images or other graphical information presented by a computing system on a display screen by using ordinary objects or even one's own hands and fingers.